Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting, and Regularization
نویسندگان
چکیده
Deep neural networks generalize well despite being exceedingly overparameterized and trained without explicit regularization. This curious phenomenon has inspired extensive research activity in establishing its statistical principles: Under what conditions is it observed? How do these depend on the data training algorithm? When does regularization benefit generalization? While such questions remain wide open for deep nets, recent works have attempted gaining insights by studying simpler, often linear, models. Our paper contributes to this growing line of work examining binary linear classification under a generative Gaussian mixture model which feature vectors take form ${{\it x}}=\pm{{\eta}}+{{\it q}}$, where mean vector $\eta$ noise q}} \sim \mathcal{N}(0,{{\Sigma}})$. Motivated results implicit bias gradient descent, we study both max-margin support machine (SVM) classifiers (corresponding logistic loss) min-norm interpolating least-squares loss). First, leverage an idea introduced [V. Muthukumar et al., arXiv:2005.08054, 2020a] relate SVM solution solution. Second, derive novel nonasymptotic bounds error latter. Combining two, present sufficient covariance spectrum signal-to-noise ratio (SNR) $SNR={||{{\eta}}||_2^4}/{{\eta}}^T{{\Sigma\eta}}$ estimators achieve asymptotically optimal performance as overparameterization increases. Interestingly, our extend noisy with constant probability flips. Contrary previously studied discriminative models, emphasize crucial role SNR interplay covariance. Finally, via combination analytical arguments numerical demonstrations identify estimator performs better than corresponding regularized estimates.
منابع مشابه
the innovation of a statistical model to estimate dependable rainfall (dr) and develop it for determination and classification of drought and wet years of iran
آب حاصل از بارش منبع تأمین نیازهای بی شمار جانداران به ویژه انسان است و هرگونه کاهش در کم و کیف آن مستقیماً حیات موجودات زنده را تحت تأثیر منفی قرار می دهد. نوسان سال به سال بارش از ویژگی های اساسی و بسیار مهم بارش های سالانه ایران محسوب می شود که آثار زیان بار آن در تمام عرصه های اقتصادی، اجتماعی و حتی سیاسی- امنیتی به نحوی منعکس می شود. چون میزان آب ناشی از بارش یکی از مولفه های اصلی برنامه ...
15 صفحه اولVolume Regularization for Binary Classification
We introduce a large-volume box classification for binary prediction, which maintains a subset of weight vectors, and specifically axis-aligned boxes. Our learning algorithm seeks for a box of large volume that contains “simple” weight vectors which most of are accurate on the training set. Two versions of the learning process are cast as convex optimization problems, and it is shown how to sol...
متن کاملVariational Mixtures of Gaussian Processes for Classification
Gaussian Processes (GPs) are powerful tools for machine learning which have been applied to both classification and regression. The mixture models of GPs were later proposed to further improve GPs for data modeling. However, these models are formulated for regression problems. In this work, we propose a new Mixture of Gaussian Processes for Classification (MGPC). Instead of the Gaussian likelih...
متن کاملthe clustering and classification data mining techniques in insurance fraud detection:the case of iranian car insurance
با توجه به گسترش روز افزون تقلب در حوزه بیمه به خصوص در بخش بیمه اتومبیل و تبعات منفی آن برای شرکت های بیمه، به کارگیری روش های مناسب و کارآمد به منظور شناسایی و کشف تقلب در این حوزه امری ضروری است. درک الگوی موجود در داده های مربوط به مطالبات گزارش شده گذشته می تواند در کشف واقعی یا غیرواقعی بودن ادعای خسارت، مفید باشد. یکی از متداول ترین و پرکاربردترین راه های کشف الگوی داده ها استفاده از ر...
ذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: SIAM journal on mathematics of data science
سال: 2022
ISSN: ['2577-0187']
DOI: https://doi.org/10.1137/21m1415121